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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S. C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )□ Responsive to communication(s) filed on . 

2a)n This action is FINAL. 2b)S This action is non-final. 

3) n Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) K Claim(s) 7-54 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) IEI Claim(s) 51-54 is/are allowed. 

6) ^ Claim(s) 1 and 36-48 is/are reiected. 

7) |EI Claim(s) 2-35. 43, 44, 49, and 50 is/are objected to. 

8) n Clajm(s) are subject to restriction and/or election requirement. 

Application Papers 

9) ^ The specification is objected to by the Examiner. 

10)^ The drawing(s) filed on 05/07/01 is/are: a)n accepted or b)S objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-1 52. 

Priority under 35 U.S.C. § 119 

12)0 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(aHd) or (f). 
a)n All b)n Some * c)\J None of: 

1 -D Certified copies of the priority documents have been received. 

2O Certified copies of the priority documents have been received in Application No. . 

3.n Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)), 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 



Drawings 



1 . The drawings are objected to because in Figure 4, block 252 the applicant fails to 
show which path leaving the block is for a "Yes" answer and which is for a "No" answer. 
Corrected drawing sheets in compliance with 37 CFR 1 .121(d) are required in reply to 
the Office action to avoid abandonment of the application. The objection to the drawings 
will not be held in abeyance. 



2. Claim 1 is provisionally rejected under the judicially created doctrine of 
obviousness-type double patenting as being unpatentable over claim 1 of copending 
Application No. 09/881,518. Although the conflicting claims are not identical, they are 
not patentably distinct from each other middleware layer recited in the above claim can 
be found in the given claims of the conflicting application. 

3. The elements of claim 1 including a middleware layer configured to facilitate 
communication between a speech-related application and a speech-related engine 
comprising: a speech component having an application interface configured to be 
coupled to the application and an engine interface coupled to the engine and one 
processing component configured to perform speech related services for the application 
and the engine are all contained in claim 1 of Application No. 09/881.518. 

Application No. 09/881,518 does not specifically mention the interfaces to be 
application-independent and engine-independent. 



Double Patenting 
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However, broadening of clainns is not grounds for patentability. 
This is a provisional obviousness-type double patenting rejection because the 
conflicting claims have not in fact been patented. 

Specification 

4. The disclosure is objected to because of the following informalities: on page 24 
line 4, the specification refers to elements 220A and 220B in Fig. 8 but these elements 
are not found in this Figure. 

Appropriate correction is required. 

Claim Objections 

5. Claims 43 and 44 are objected to because of the following informalities: claim 43 
depends on succeeding claim 44, which in turn depends on claim 43. Also claim 44 
should be changed from "The multi-process speech recognition middleware claim 43 
layer of to read -The multi-process speech recognition middleware layer of claim 43- 
, or whatever dependence was intended. 

Appropriate correction is required. 

Claim Rejections - 35 USC §112 

6. The following is a quotation of the second paragraph of 35 U.S.C. 112: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 
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7. Claim 36 recites the limitation "the SR engine" in line 5. There is insufficient 
antecedent basis for this limitation in the claim. Claim 1 does not make any previous 
mention of a SR engine. 

Claim Rejections - 35 USC § 102 

8. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent 
granted on an application for patent by another filed in the United States before the invention by the 
applicant for patent, except that an international application filed under the treaty defined in section 
351(a) shall have the effects for purposes of this subsection of an application filed in the United States 
only if the international application designated the United States and was published under Article 21(2) 
of such treaty in the English language. 

9. Claims 40-46 are rejected under 35 U.S.C. 102(e) as being anticipated by Hunt 
et al. (U.S. Pat, 6,374,226). 

Regarding claim 40, Hunt teaches a multi-process speech recognition 
middleware layer to facilitate communication between a speech recognition engine and 
one or more applications, the middleware layer comprising: 

a first and second process (Speech Controller Modules A and B, Fig. 1 , elements 
15a and 15b); 

each with a context object having an application interface to enable application 
control of a first plurality of attributes of the speech recognition and an engine interface 
(speech controller object is responsive to the program component, col. 3, lines 41-53); 
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each with a grammar object having an application interface and an engine 
interface and storing a first grammar used by the corresponding process (application 
interface col. 4, lines 32-37 and engine interface col. 4, lines 62-65); and 

a server process configured to receive result information provided by the SR 
engine and prove the result information to the first or second process, to which the 
result information belongs (resultant information is returned from the SR engine to the 
appropriate process, col. 4, lines 19-27). 

10. As per claim 41 , Hunt teaches the first and second grammars each include a 
plurality of rules and a grammar engine configured to store a grammar indication 
indicating the grammar to which each of the plurality of rules belong (each rule is stored 
in a table with other rules from the same grammar and the table is given a grammar 
name, Fig. 3, element 72). 

11. As per claim 42, Hunt teaches when the SR engine returns the result information 
it also returns a rule identifier identifying a rule which spawned the result information 
(speech recognizer identifies the rule and the speech controller module parses the 
resultant information based up this identified rule so an identifier would inherently be 
passed from the speech recognizer to the speech controller to indicate this information, 
col. 6, lines 17-44). 

12. As per claim 43, Hunt teaches the grammar engine examines the rule identifier to 
determine a particular grammar to which the identified rule belongs (speech recognizer 
identifies rule and the speech controller module parses the resultant information based 
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up this identified rule so the middleware would inherently determine the rule identified 
by the identifier, col. 6, lines 17-44). 

13. As per claim 44, Hunt teaches that the server queries the first or second 
grammar object containing the particular grammar to receive an identity of an 
associated context object (passes parse tree to an object associated with the root rule 
and so it must first identify this object before it passes the information, col. 6, lines 45- 
46). 

14. As per claim 45, Hunt teaches the server process notifies the associated context 
object that the result belongs to its associated process (passes parse tree to an object 
associated with the root rule and so it notifies the object by passing it the information, 
col. 6, lines 45-46). 

15. As per claim 46, Hunt teaches the SR engine returns preliminary information to 
the server process and wherein the server process is configured to notify the first and 
second context objects of the preliminary information (returns result information to the 
speech controller and suggests updating the result information afterwards hence the 
original result information would be preliminary information, col. 14, lines 7-13). 

Claim Rejections - 35 USC § 103 

16. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 
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17. Claim 47 and 48 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Hyde-Thomson et al. (U.S. Pat. 6,487,533) in view of Didcock (U.S. Pat. 6,466,909). 

Hyde-Thomson teaches a middleware layer configured to facilitate 
communication between one or more applications and a plurality of text-to-speech 
engines (Fig. 4, elements 242-244) comprising: 

multi-voice speech synthesis (different languages suggests different voices, col. 
5, lines 10-16); 

a first voice object configured to receive information from the application (voice 
message review request, col, 4, lines 49-59); 

instantiate a first and second TTS engine (engines loaded into memory when a 
different language is required, col. 5, lines 16-20); 

receive a speak request requesting at least one of the TTS engines to speak a 
message (voice message review request, col. 4, lines 49-59); and 

the first voice object has an engine interface to call a specified one of the first 
and second TTS engines to synthesize input data (message inquiry unit selects the TTS 
engine based upon the language identifier, col. 7, lines 13-16). 

Hyde-Thomson does not teach the first voice object receiving TTS engine 
attribute information and using this information to instantiate the engine. 

However, the Examiner takes Official Notice that common knowledge in the art to 
allow the user to select the attributes for the speech engine rather than automatically 
generating this information and it would have been obvious to one of ordinary skill in the 
art at the time of invention to modify the system of Hyde-Thomson to have the first voice 
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object receive TTS engine attribute information and using this information to instantiate 
the engine because the synthesized speech would be more to the specific liking of the 
user. 

Hyde-Thompson does not teach the TTS engines to receive priority information 
associated with each speak request indicative of a precedence each speech request is 
to take, 

Didcock teaches a TTS engine where the requests are queued according to 
different priorities and hence the engine would receive information associated with 
these queues in order to process them correctly (col. 3, lines 44-50). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify the system of Hyde-Thomson to have the TTS engines receive 
priority information associated with each speak request indicative of a precedence each 
speech request is to take as taught by Didcock because it would ensure that the more 
important requests are processed first hence allowing the system to control the flow of 
user requests. 

18. As per claim 48, Hyde-Thomson teaches the first voice object is configured to 
receive a normal priority associated with a message and to call the TTS engines so the 
message with normal priority is spoken in turn (message is buffered before synthesizing 
so that all messages are given the same normal priority, col. 6, lines 36-42). 



Allowable Subject Matter 

19. Claims 51-54 are allowed. 
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The following is an examiner's statement of reasons for allowance: Claim 51 is 
found to be allowable over prior art because it recites the combination of a middleware 
layer that comprises both an application-independent interface configured to be coupled 
to an application and an engine-independent interface configured to be coupled to an 
engine that facilitate communication between a speech-related application and a 
speech-related engine. 

Comerford et al, (U.S. Pat. 6,513,009) teaches a middleware layer configured to 
facilitate communication between a speech-related application and a speech-related 
engine that has an application interface and engine-independent interface but does not 
teach the application interface to be application independent. 

Hunt teaches a middleware layer configured to facilitate communication between 
a speech-related application and a speech-related engine that has an application- 
independent interface and engine interface but does not teach the engine interface to 
be engine-independent. 

It would not have been obvious to one of ordinary skill in the art at the time of 
invention to combine these references because neither teaches nor fairly suggests their 
combination. 

Claim 53 is allowed over prior art because it recites the combination of obtaining 
the data formats used by both the speech engine and the audio device and if the 
formats are not consistent, attempting to change the data format of the data used by at 
least one of the engine and the audio device. 
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Johnson (U.S. Pat. 5,748,974) teaches a method of formatting data for use by a 
speech engine and an audio device that obtains the data format used by the engine 
and the audio device (response generator generates an output based upon the 
application and would inherently check the data formats of the speech engine and the 
chosen output device to see if they are compatible, col. 4, lines 48-54). 

However, Johnson does not teach attempting to change the data format of the 
data used by at least one of the engine and the audio device if the engine data format 
and the audio data format are not consistent. 

Dependent claims 52 and 54 are allowed because they further limit the subject 
matter of their parent claims. 

20. Claim 1 would be allowable once the double-patenting issue has been resolved 
for the same reasons as for claim 51 . 

21 . Claims 2-35 are objected to as being dependent upon a rejected base claim, but 
would be allowable if rewritten in independent form including all of the limitations of the 
base claim and any intervening claims. 

22. Claims 49 and 50 are objected to as being dependent upon a rejected base 
claim, but would be allowable if rewritten in independent form including all of the 
limitations of the base claim and any intervening claims. 

23. Claim 49 would be allowable over prior art because it recites the combination 
wherein the first voice object receives a speakover priority associated with a message 
and to call the TTS engines so the message with speakover priority is spoken at a same 
time as other currently speaking messages. 
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Didcock teaches a multi-voice speech synthesis middleware layer that queues its 
messages with normal and alert priorities (col. 4, lines 6-1 1 and 36-44) but does not 
teach a speakover priority. 

Dependent claim 50 would be allowable because it further limits the subject 
matter of its parent claim. 

24. Claim 36 would be allowable if rewritten or amended to overcome the rejection(s) 
under 35 U.S.C. 112, 2nd paragraph, set forth in this Office action. 

25. Claims 37-39 would be allowable if rewritten to overcome the rejection(s) under 
35 U.S.C. 1 12, 2nd paragraph, set forth in this Office action and to include all of the 
limitations of the base claim and any intervening claims. 

Conclusion 

26. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. Hataoka et al. ("Sophisticated Speech Processing Middleware on 
Microprocessor"), Baker et al. (U.S. Pat. 6,456,974), Profit etal. (U.S. Pat. 6,636,831), 
and Coffman et al. (U.S. Pat. 6,377,913) teach middleware for application-independent 
interface. Greene et al. (U.S. Pat. 6,377,925) teaches a middleware for engine- 
independent interface. Peres et al. (U.S. Pat. 6,618,703), Mitchell et al. (U.S. Pat. 
5,799,273). Abella et al. (U.S. Pat. 6,044,347), and Wilson (U.S. Pat. 6,526,381), teach 
a system with a middleware to control the interaction between a speech application and 
engine. 
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Any inquiry concerning tills communication or earlier communications from the 
examiner sliould be directed to IVIattliew J Sked whose telephone number is (703) 305- 
8663. The examiner can normally be reached on Mon-Fri (8:00 am - 4:30 pm). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Talivaldis Smits can be reached on (703) 306-301 1 . The fax phone number 
for the organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 
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